Virtual camera control using motion control systems for augmented reality

ABSTRACT

There is provided a system and method for integrating a virtual rendering system and a motion control system to provide an augmented reality. There is provided a method for integrating a virtual rendering system and a motion control system for outputting a composite render to a display, the method comprising obtaining, from the motion control system, a robotic camera configuration of a robotic camera in a real environment, programming the virtual rendering system using the robotic camera configuration to correspondingly control a virtual camera in a virtual environment, obtaining a virtually rendered feed using the virtual camera, capturing a video capture feed using the robotic camera, rendering the composite render by processing the feeds, and outputting the composite render to the display.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to digital video. Moreparticularly, the present invention relates to digital video rendering.

2. Background Art

Modern commodity PC hardware and videogame consoles are often equippedwith sufficient processing capability to enable high-resolutionreal-time three-dimensional graphics rendering. Even portable devicessuch as mobile phones and handheld gaming systems are often equippedwith scaled down real-time three-dimensional graphics support. Suchlow-cost commodity graphics processing hardware has enabled a widevariety of entertainment and productivity applications to supportenhanced visual presentations for greater user engagement and enjoyment.

In particular, real-time three-dimensional graphics rendering has founditself as a highly supportive role in live broadcast programming. Forexample, coverage of sports and other live events can be readilyenhanced with composite renderings using three-dimensional graphics foralternative or substitute object rendering, strategy simulations,information boxes, alternative viewpoints, and other effects. Althoughthree-dimensional analysis tools exist for composite rendering, suchtools cannot be used to accurately replay prior camera paths usingmanually controlled camera systems. As a result, important past eventssuch as a winning play or a brilliant strategy cannot be effectivelyanalyzed. Furthermore, manually controlled camera systems are difficultto synchronize with virtual environment camera systems, as the cameraoperator cannot perceive the virtual environment. Absent suchsynchronization, it is difficult to create acceptable compositerendering within real-time constraints. As a result, viewer engagementis low since analysis is limited to imprecise and unsynchronizedmanually controlled camera movements.

Accordingly, there is a need to overcome the drawbacks and deficienciesin the art by providing a way to create composite renderings using livefootage with real-time three-dimensional graphics rendering for highviewer impact and engagement.

SUMMARY OF THE INVENTION

There are provided systems and methods for integrating a virtualrendering system and a motion control system to provide an augmentedreality, substantially as shown in and/or described in connection withat least one of the figures, as set forth more completely in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present invention will become morereadily apparent to those ordinarily skilled in the art after reviewingthe following detailed description and accompanying drawings, wherein:

FIG. 1 presents a system for integrating a virtual rendering system anda motion control system to provide an augmented reality, according toone embodiment of the present invention;

FIG. 2 presents a diagram of a virtual camera path configured to match arobotic camera path used by a motion control system, according to oneembodiment of the present invention;

FIG. 3 presents a diagram of a composite render being generated,according to one embodiment of the present invention; and

FIG. 4 shows a flowchart describing the steps, according to oneembodiment of the present invention, by a virtual rendering system and amotion control system may be integrated for outputting a compositerender of an augmented reality to a display.

DETAILED DESCRIPTION OF THE INVENTION

The present application is directed to a system and method forintegrating a virtual rendering system and a motion control system toprovide an augmented reality. The following description containsspecific information pertaining to the implementation of the presentinvention. One skilled in the art will recognize that the presentinvention may be implemented in a manner different from thatspecifically discussed in the present application. Moreover, some of thespecific details of the invention are not discussed in order not toobscure the invention. The specific details not described in the presentapplication are within the knowledge of a person of ordinary skill inthe art. The drawings in the present application and their accompanyingdetailed description are directed to merely exemplary embodiments of theinvention. To maintain brevity, other embodiments of the invention,which use the principles of the present invention, are not specificallydescribed in the present application and are not specificallyillustrated by the present drawings.

FIG. 1 presents a system for integrating a virtual rendering system anda motion control system to provide an augmented reality, according toone embodiment of the present invention. Diagram 100 of FIG. 1 includesvirtual rendering system 110, video capture system 130, mastercontroller 150, composite render 155, live broadcast link 156, anddisplay 157. Virtual rendering system 110 includes rendering enginecontroller 111, auxiliary rendering engine 112, slave rendering engines113 a-113 b, and virtual environment 120. Virtual environment 120includes virtual cameras 121 a-121 b. Virtual camera 121 a includes data122 a. Virtual camera 121 b includes data 122 b. Video capture system130 includes camera motion controller 131 and real environment 140. Realenvironment 140 includes robotic cameras 141 a-141 b. Robotic camera 141a includes data 142 a. Robotic camera 141 b includes data 142 b.

Rendering engine controller 111, auxiliary rendering engine 112, slaverendering engine 113 a, and slave rendering engine 113 b may eachexecute on several separate servers, each comprising standard commodityPC hardware or a videogame console system. Alternatively, the engines ofvirtual rendering system 110 may be consolidated into a single server,or distributed remotely across a network to minimize the amount ofnecessary on-site hardware. Rendering engine controller 111 maycoordinate control and data sharing between the different renderingsubsystems, as shown in FIG. 1. Auxiliary rendering engine 112 mayprovide static graphics overlays and other graphical effects that do notrequire input from virtual environment 120. Slave rendering engines 113a-113 b each control virtual cameras 121 a-121 b respectively to receivea virtually rendered feed of virtual environment 120.

Real environment 140 corresponds to an actual physical environmentrepresented by virtual environment 120. For example, real environment140 might comprise an indoor or outdoor sports field or stadium, a golfcourse, a natural environment, an urban environment, or any otherlocale. Besides sports entertainment environments, other environmentsfor focuses such as educational or informational applications may alsobe used. Virtual environment 120 may then be created using manualthree-dimensional environmental modeling, automated photographic orvideo extrapolation, or some other manual or automated method.

Data 142 a-142 b describing the configuration of robotic cameras 141a-141 b within real environment 140 may each include, for example,position data such as three-dimensional coordinates, camera field ofview orientation data such as camera angle, focal length and focusdistance, movement data such as a motion path or velocity andacceleration, and camera characteristics such as lens parameters, camerasize, center of lens, and other camera model details. Three-dimensionalcoordinates between virtual environment 120 and real environment 140 maybe defined using a common frame of reference, such as setting aparticular corner of a field or a particular landmark as a common (0, 0,0) coordinate. The motion path may then describe the changing of theabove data parameters with respect to time, such as thethree-dimensional coordinates with respect to time or the camera fieldof view with respect to time.

Master controller 150 may receive data 142 a-142 b from camera motioncontroller 131 of video capture system 130. Camera motion controller 131may, for example, query robotic cameras 141 a-141 b to retrieve data 142a-142 b, respectively. The motion path data in data 142 a-142 b may bedictated by manual control such as by a camera operator, by tracking themotion of a particular object or focusing on a particular scene ineither virtual environment 120 or real environment 140, by replaying apreviously recorded path of movement or another predetermined path, by avideo analysis using video capture system 130, or by using some othercriteria. Tracked objects may include, for example, a ball or aparticipating player of a game such as a sports match, and may bevirtual or real. Thus, robotic cameras 141 a-141 b can function asmotion control cameras. Camera motion controller 131 might also acceptuser input to modify the motion paths contained in data 142 a-142 b.

Master controller 150 may then direct virtual rendering system 110 tocontrol virtual cameras 121 a-121 b according to the retrieved motionpath data from data 142 a-142 b. Once the virtual cameras are properlyconfigured by appropriately programming the motion paths of data 122a-122 b, master controller 150 may then query virtual rendering system110 for virtually rendered feeds and video capture system 130 for videocapture feeds. Master controller 150 may then act as a renderingcontroller by combining the feeds smoothly using standard broadcast keytechnology such as chroma key or key/fill to generate composite render155, which includes real and virtual feed elements arranged in aspecific desired manner for broadcast over live broadcast link 156 todisplay 157. Live broadcast link 156 may comprise, for example, asatellite uplink to a television studio, from where the broadcastedmaterial is disseminated to the general public. Display 157 may thenrepresent, for example, a television of a viewer watching the broadcast.

Although virtual rendering system 110 of FIG. 1 depicts only two slaverendering engines each controlling exactly one virtual camera,alternative embodiments may use any arbitrary number of slave renderingengines to control any arbitrary number of virtual cameras. Morespecifically, each slave rendering engine may control more than onevirtual camera. Similarly, although video capture system 130 of FIG. 1only depicts two robotic cameras, alternative embodiments may use anyarbitrary number of robotic cameras to be controlled by camera motioncontroller 131 of video capture system 130. In this manner, thecomposite rendering system shown in FIG. 1 can be scaled to as manycamera angles and viewpoints as desired, in either virtual environment120 or real environment 140.

FIG. 2 presents a diagram of a virtual camera path configured to match arobotic camera path used by a motion control system, according to oneembodiment of the present invention. Diagram 200 of FIG. 2 includesvirtual environment 220 and real environment 240. Virtual environment220 includes virtual camera 221 and virtual camera path 223. Realenvironment 240 includes robotic camera 241, robotic camera path 243,real object 245, and real object path 246. With regards to FIG. 2, itshould be noted that virtual environment 220 corresponds to virtualenvironment 120 from FIG. 1 and that real environment 240 corresponds toreal environment 140 from FIG. 1. Moreover, although FIG. 2 only depictsa single virtual camera and a single robotic camera for simplicity,alternative embodiments may use multiple virtual cameras and multiplerobotic cameras.

As previously discussed in FIG. 1, master controller 150 may directvideo capture system 130 to control virtual cameras according to motionpaths provided by a motion control system. FIG. 2 shows an example ofthis manner of control, where robotic camera 241, comprising a motioncontrol camera such as a gantry supported fly-by camera, is programmedto record a path of movement. For example, robotic camera 241 may beprogrammed to focus on the movement of real object 245, a football,following real object path 246, a long distance pass. Thus, roboticcamera 241 may follow robotic camera path 243, with camera orientationfollowing real object path 246 as indicated by the dotted arrows.Robotic camera path 243 may then be recorded and programmed into virtualcamera 221 of virtual environment 220, so that virtual camera 221 canfollow virtual camera path 223 mirroring robotic camera path 243.

As shown in FIG. 2, the camera orientation of virtual camera 221 movesas if it were following real object path 246 within virtual environment220, even though there is no corresponding virtual object for realobject 245 in virtual environment 220. By using the system describedabove in FIG. 1, virtual camera 221 can thus be synchronized to thecamera movements of robotic camera 241. Composite rendering of real andvirtual environments, also known as “augmented reality”, is thusfacilitated, as the camera views in virtual environment 220 and realenvironment 240 can be matched according to any desired camera path,opening up limitless possibilities for dynamic camerawork. Previouslyrecorded camera paths may also be recorded into storage for futurereplay, both by robotic cameras and virtual cameras.

The example shown in FIG. 2 might be used, for example, to present adynamic panning camera view showing the long distance football passdefined by real object path 246 according to several virtual scenarios.For example, players might be virtually positioned and simulated invarious scenarios to determine whether an alternative strategy mighthave resulted in an interception of the pass. Thus, a composite rendermight show a real sports field background and the actual football, orreal object 245, in a video feed captured from real environment 240, butwith virtual players rendered in virtual environment 220. Thus, thecomposite render can provide a realistic camera fly-by with the footballand background elements from real environment 240 and virtual playersrendered from virtual environment 220. Variations of this procedure maybe used to present various hypothetical plays and strategy analyses in arealistic and engaging manner for high viewer impact.

FIG. 3 presents a diagram of a composite render being generated,according to one embodiment of the present invention. Diagram 300 ofFIG. 3 includes virtual rendering system 310, virtually rendered feeds315 a-315 b, video capture system 330, video capture feeds 335 a-335 b,master controller 350, composite render 355, live broadcast link 356,and display 357. With regards to FIG. 3, it should be noted that virtualrendering system 310 corresponds to virtual rendering system 110 fromFIG. 1, that video capture system 330 corresponds to video capturesystem 130, that master controller 350 corresponds to master controller150, that composite render 355 corresponds to composite render 155, thatlive broadcast link 356 corresponds to live broadcast link 156, and thatdisplay 357 corresponds to display 157.

As shown in FIG. 3, virtual rendering system 310 provides mastercontroller 350 with virtually rendered feeds 315 a-315 b, while videocapture system 330 provides master controller 350 with video capturefeeds 335 a-335 b. For example, video capture feed 335 a mightcorrespond to a feed generated by robotic camera 241 in FIG. 2, whereasvirtually rendered feed 315 a might correspond to a feed generated byvirtual camera 221 in FIG. 2. Virtually rendered feed 315 b maycorrespond to a feed created by an overhead virtual camera providing abird's eye overview of virtual environment 220 from FIG. 2, whereasvideo capture feed 335 b may correspond to a feed created by an overheadrobotic camera providing a bird's eye overview of real environment 240from FIG. 2.

Master controller 350 may then combine virtually rendered feed 315 a andvideo capture feed 335 a for an augmented reality fly-by scene and alsocombine virtually rendered feed 315 b and video capture feed 335 b foran augmented reality bird's eye overview scene. As previously discussed,master controller 350 may use standard broadcast key technologies tocombine the different feeds smoothly so that the juxtaposition of realand virtual elements is visually unobtrusive. Master controller 350 maythen use these combined scenes in composite render 355 through variouspresentation methods such as split screen, cascaded or tiled frames,“picture-in-picture”, three-dimensional surfaces, and other formattedlayouts. Master controller 350 may then forward composite render 355over live broadcast link 356 for showing on display 357.

Master controller 350 may repeat the above process of generatingcomposite render 355 in a periodic manner, such as 24, 30, or 60 timesper second or higher in order to accommodate a desired videobroadcasting framerate.

Although FIG. 3 only shows a single composite render 355, alternativeembodiments may use several composite renders. For example, mastercontroller 350 may generate multiple composite renders to providedifferent camera views for multiple broadcast channels, to customizebased on a target broadcast region or audience demographics, to focus ona particular team in a sports match, or to support any otherbroadcasting application that may require multiple concurrent videostreams. By adding additional slave rendering engines and roboticcameras, augmented reality rendering systems can be readily scaled andconfigured to support large-scale projects.

FIG. 4 shows a flowchart describing the steps, according to oneembodiment of the present invention, by a virtual rendering system and amotion control system may be integrated for outputting a compositerender of an augmented reality to a display. Certain details andfeatures have been left out of flowchart 400 that are apparent to aperson of ordinary skill in the art. For example, a step may compriseone or more substeps or may involve specialized equipment or materials,as known in the art. While steps 410 through 460 indicated in flowchart400 are sufficient to describe one embodiment of the present invention,other embodiments of the invention may utilize steps different fromthose shown in flowchart 400.

Referring to step 410 of flowchart 400 in FIG. 4 and diagram 100 of FIG.1, step 410 of flowchart 400 comprises obtaining, from camera motioncontroller 131 of video capture system 130, data 142 a of robotic camera141 a in real environment 140. Camera motion controller 131 may thenretrieve data 142 a from robotic camera 141 a for relaying back tomaster controller 150. As previously described, data 142 a may containvarious information concerning the configuration of robotic camera 141 asuch as three-dimensional position and movement, camera focus and view,camera model parameters, and other details.

Referring to step 420 of flowchart 400 in FIG. 4 and diagram 100 of FIG.1, step 420 of flowchart 400 comprises programming virtual renderingsystem 110 using data 142 a obtained from step 410 to correspondinglycontrol virtual camera 121 a in virtual environment 120. For example,master controller 150 may instruct rendering engine controller 111 toprogram values into data 122 a to match data 142 a as closely aspossible. As previously discussed, data 142 a may includethree-dimensional coordinates and camera field of view with respect totime. Assuming virtual camera 221 and robotic camera 241 correspond tovirtual camera 121 a and robotic camera 141 a, the result of settingdata 122 a to match data 142 a may be manifested by virtual camera path223 mimicking robotic camera path 243, as shown in FIG. 2.

Referring to step 430 of flowchart 400 in FIG. 4, diagram 200 of FIG. 2,and diagram 300 of FIG. 3, step 430 of flowchart 400 comprisesobtaining, from virtual rendering system 310, virtually rendered feed315 a of virtual environment 220 using virtual camera 221. Since themotion path of virtual camera 221 was previously programmed in step 420,step 430 results in master controller 350 receiving virtually renderedfeed 315 a comprising fly-by footage according to virtual camera path223.

Referring to step 440 of flowchart 400 in FIG. 4, diagram 200 of FIG. 2,and diagram 300 of FIG. 3, step 440 of flowchart 400 comprisesobtaining, from video capture system 330, video capture feed 335 a ofreal environment 240 using robotic camera 241. As previously discussed,robotic camera path 243 may be defined in any number of ways, such as bymanual control, object tracking, recorded motion replay, orpredetermined paths. As shown in FIG. 2, robotic camera path 243 isdefined as an arc with the camera field of view following real object245 as it progresses through real object path 246. Thus, mastercontroller 350 may receive video capture feed 335 a comprising fly-byfootage according to robotic camera path 243, wherein the feed includesa rendering of real object 245.

Referring to step 450 of flowchart 400 in FIG. 4 and diagram 300 of FIG.3, step 450 of flowchart 400 comprises rendering composite render 355 byprocessing video capture feed 335 a from step 440 and virtually renderedfeed 315 a from step 430. As previously discussed, master controller 350may accomplish step 450 using standard broadcast key technology such aschroma key or key/fill techniques to isolate and combine components fromeach feed to produce a result with a smooth visual juxtaposition of realand virtual elements.

Referring to step 460 of flowchart 400 in FIG. 4 and diagram 300 of FIG.3, step 460 of flowchart 400 comprises outputting composite render 355from step 450 to display 357. As shown in FIG. 3, master controller 350may send composite render 355 using live broadcast link 356, which mightcomprise a satellite uplink to a broadcast station for publicdissemination. Eventually, composite render 355 shows on display 357,which might comprise the television of a viewer at home.

While the above steps 410-460 have been described with respect to asingle virtual camera, a single robotic camera, and a single compositerender, steps 410-460 may also be repeated as necessary to supportmultiple virtual cameras, multiple robotic cameras, and multiplecomposite renders, as previously described. In this manner, thedescribed rendering system can be flexibly scaled to larger projects byincreasing the number of slave rendering systems and robotic cameras tohandle additional feeds in real-time.

In this manner, live events such as sports can be enhanced withhigh-impact augmented reality segments by leveraging the cost effectivereal-time graphical capabilities of modern commodity PC hardware andgame consoles. This can provide networks with a competitive advantage bydrawing in and retaining greater viewership by providing compellingaugmented reality content while requiring only minor additionalinfrastructure outlays over standard rendering systems. Since commodityhardware parts are used and numerous effective virtual rendering systemsand engines are available for licensing, expensive proprietary systemsand vendor lockout may be avoided, further reducing total cost ofownership.

From the above description of the invention it is manifest that varioustechniques can be used for implementing the concepts of the presentinvention without departing from its scope. Moreover, while theinvention has been described with specific reference to certainembodiments, a person of ordinary skills in the art would recognize thatchanges can be made in form and detail without departing from the spiritand the scope of the invention. As such, the described embodiments areto be considered in all respects as illustrative and not restrictive. Itshould also be understood that the invention is not limited to theparticular embodiments described herein, but is capable of manyrearrangements, modifications, and substitutions without departing fromthe scope of the invention.

What is claimed is:
 1. A method for integrating a virtual renderingsystem and a motion control system to output a composite render to adisplay, the method comprising: obtaining, from the motion controlsystem, a first camera configuration, wherein the first cameraconfiguration includes camera position data, camera field of vieworientation data, camera movement data and camera lens characteristicsdata; programming the virtual rendering system using the first cameraconfiguration to correspondingly control a first virtual camera in avirtual environment; obtaining, from the virtual rendering system, afirst virtually rendered feed of the virtual environment using the firstvirtual camera; capturing, from a video capture system, a first videocapture feed of a real environment using the first camera configuration;rendering the composite render, based on a target broadcast region, byprocessing the first virtually rendered feed and the first video capturefeed, wherein the composite render includes a real tracked object and atleast one virtual player in the virtual environment; and outputting thecomposite render to the display.
 2. The method of claim 1, wherein thefirst motion path includes a three-dimensional position with respect totime.
 3. The method of claim 1, wherein the first motion path includes acamera orientation or field of view with respect to time.
 4. The methodof claim 1, wherein the camera lens characteristics data includes cameralens parameters.
 5. The method of claim 1, wherein the first motion pathis based on a predetermined path.
 6. The method of claim 1, wherein thefirst motion path is based on a previously recorded path.
 7. The methodof claim 1, wherein the first motion path is based on a video analysisusing the video capture system.
 8. The method of claim 1 furthercomprising prior to the programming of the virtual rendering system,modifying the first motion path according to a user input.
 9. The methodof claim 1 further comprising prior to the rendering of the compositerender: obtaining, from the motion control system, a second cameraconfiguration including a second motion path in the real environment;programming the virtual rendering system using the second cameraconfiguration to correspondingly control a second virtual camera in thevirtual environment; obtaining, from the virtual rendering system, asecond virtually rendered feed of the virtual environment using thesecond virtual camera; capturing, from a video capture system, a secondvideo capture feed of the real environment using the second cameraconfiguration; wherein the rendering of the composite render furtherprocesses the second virtually rendered feed and the second videocapture feed.
 10. A rendering controller for outputting a compositerender to a display, the rendering device comprising: a processorconfigured to: obtain, from a motion control system, a first cameraconfiguration, wherein the first camera configuration includes cameraposition data, camera field of view orientation data, camera movementdata and camera lens characteristics data; program a virtual renderingsystem using the first camera configuration to correspondingly control afirst virtual camera in a virtual environment; obtain, from the virtualrendering system, a first virtually rendered feed of the virtualenvironment using the first virtual camera; capture, from a videocapture system, a first video capture feed of a real environment usingthe first camera configuration; render the composite render, based on atarget broadcast region, by processing the first virtually rendered feedand the first video capture feed, wherein the composite render includesa real tracked object and at leastone virtual player in the virtualenvironment; and output the composite render to the display.
 11. Therendering controller of claim 10, wherein the first motion path includesa three-dimensional position with respect to time.
 12. The renderingcontroller of claim 10, wherein the first motion path includes a cameraorientation or field of view with respect to time.
 13. The renderingcontroller of claim 10, wherein the camera lens characteristics dataincludes camera lens parameters.
 14. The rendering controller of claim10, wherein the first motion path is based on a predetermined path. 15.The rendering controller of claim 10, wherein the first motion path isbased on a previously recorded path.
 16. The rendering controller ofclaim 10, wherein the first motion path is based on a video analysisusing the video capture system.
 17. The rendering controller of claim10, wherein prior to the programming of the virtual rendering system,the processor is configured to modify the first motion path according toa user input.
 18. The rendering controller of claim 10, wherein prior tothe rendering of the composite render the processor is configured to:obtain, from the motion control system, a second camera configurationincluding a second motion path in the real environment; program thevirtual rendering system using the second camera configuration tocorrespondingly control a second virtual camera in the virtualenvironment; obtain, from the virtual rendering system, a secondvirtually rendered feed of the virtual environment using the secondvirtual camera; and capture, from a video capture system, a second videocapture feed of the real environment using the second cameraconfiguration; wherein the processor is configured to render thecomposite render by further processing the second virtually renderedfeed and the second video capture feed.